Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DM-40434: Prompt Processing continues processing after timeout or termination #182

Merged
merged 9 commits into from
Jul 22, 2024

Conversation

kfindeisen
Copy link
Member

@kfindeisen kfindeisen commented Jul 15, 2024

This PR adds configurable timeouts and grace periods to the service container, as well as a handler for TERM signals sent by Knative.

Closes #91.

These timeouts make it possible to abort processing instead of having
to wait until it finishes (as is the case with a Knative timeout).
Optional is deprecated, and typing.Tuple can be replaced with tuple.
@kfindeisen kfindeisen force-pushed the tickets/DM-40434 branch 2 times, most recently from ba05510 to 8e3b703 Compare July 18, 2024 20:43
This interrupt can be used to indicate that the app should clean up,
then shut down.
This spec is already assumed by the pipeline, and without explicit
documentation it's not obvious why no current branch raises
RetriableError.
Factoring out the check makes it easier to handle different exceptions
slightly differently.
Copy link
Collaborator

@dspeck1 dspeck1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

If pipeline processing is interrupted (the most likely case),
retriability is handled with almost the same logic as pipeline errors
(the exception is that an interrupt is our first nondeterministic
failure mode). Otherwise, the failure is assumed non-retriable just
in case.
This makes it possible to adjust the grace period to accommodate the
service's long shutdown time (in particular, syncing to the
central repo).
This handler lets us retry services that were shut down during
prep_butler or while waiting for raws. The outermost handler is still
needed to catch interrupts that slip through the cracks.
@kfindeisen kfindeisen merged commit 7a1fb03 into main Jul 22, 2024
6 checks passed
@kfindeisen kfindeisen deleted the tickets/DM-40434 branch July 22, 2024 22:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants